Image display apparatus and image display method

ABSTRACT

A display image is superimposed and displayed on an outside image in a preferred manner. 
     An optical system superimposes a display image displayed on a display device onto an outside image, and leads the display image to an eye of an observer. A display control unit controls the display size and the display position of the display image on the display device so that the display image is displayed in an image superimposition region (a flat region) detected from the outside image. For example, the display control unit controls the display state of the display image in accordance with the state of the image superimposition region in the outside image. Also, the display control unit performs control to selectively display the display image in a line-of-sight region or outside the line-of-sight region in accordance with the line of sight of the observer.

TECHNICAL FIELD

The present technique relates to image display apparatuses and imagedisplay methods, and more particularly, to an image display apparatussuch as an optically-transmissive head mount display designed tosuperimpose an image displayed on a display device onto an outside imageand lead the image to an eye of an observer, and the like.

BACKGROUND ART

Head mount displays (HMDs) that are mounted on the heads of users havebecome known in recent years. A head mount display, in principle, isdesigned to enlarge an image displayed on a small-sized display devicewith an enlarging optical system, and lead the image to an eye of anobserver. That is, a head mount display is designed to optically enlargean image displayed on a display device, and allow a user to observe theimage as an enlarged virtual image.

As this type of head mount display, an optically-transmissive head mountdisplay designed to enable an observer to observe not only the abovementioned virtual image but also an outside image is known. Thisoptically-transmissive head mount display is designed to superimpose animage displayed on a display device onto an outside image and lead theimage to an eye of an observer with an optical system.

The visibility of a virtual image reproduced by thisoptically-transmissive head mount display depends on the environment inwhich this virtual image is displayed. For example, inconsistencybetween the display state of this virtual image and the state of thereal world hinders comfortable observation, or the visibility of thevirtual image is lowered depending on the display position thereof.

Patent Document 1 discloses adjusting the depth of an entire image bytaking into account to which object the observer is paying attention,for example. Patent Document 2 discloses adjusting the disparity of avirtual image in accordance with convergence of the eyes by using aneye-gaze tracking technique, for example.

CITATION LIST Patent Document

Patent Document 1: JP 05-328408 A

Patent Document 2: JP 10-188034 A

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

By the techniques disclosed in Patent Documents 1 and 2, the position inwhich a virtual image is displayed, and the size of the virtual imageare not taken into consideration. Furthermore, the techniques disclosedin Patent Documents 1 and 2 do not teach display of a stereoscopic (3D)image as a virtual image.

The present technique aims to superimpose and display a display image onan outside image in a preferred manner.

Solutions to Problems

A concept of the present technique lies in an image display apparatusthat includes:

an optical system that superimposes a display image displayed on adisplay device onto an outside image, and leads the display image to aneye of an observer; and

a display control unit that controls a display size and a displayposition of the display image on the display device so that the displayimage is displayed in an image superimposition region detected from theoutside image.

In the present technique, the image displayed on the display device issuperimposed on the outside image and is led to an eye of an observer bythe optical system. In a case where the optical system is an enlargingoptical system, the image displayed on the display device is opticallyenlarged, and is observed as an enlarged virtual image by the observer.

The display control unit controls the display size and the displayposition of the display image to be superimposed and displayed on theoutside image. In this case, the display size and the display positionof the display image on the display device are controlled so that thedisplay image is displayed in an image superimposition region detectedfrom the outside image.

The image superimposition region is detected based on captured imagedata obtained by forming the outside image, for example. In this case, aflat region included in the outside image is detected as the imagesuperimposition region, for example. This detection of the imagesuperimposition region may be performed in a cloud, for example.

The display control unit may control the display size and the displayposition of the display image by processing (geometrically transforming)image data for displaying the display image on the display device basedon information about the image superimposition region, for example. Inthis case, the display size and the display position of the displayimage are electronically controlled, and such control is easier.

In the present technique described above, display control is performedso that the display image is displayed in the image superimpositionregion such as a flat region detected from the outside image.Accordingly, it becomes easier for the observer to visually recognizethe display image superimposed and displayed on the outside image.

In the present technique, the display control unit may control thedisplay state of the display image in accordance with the state of theimage superimposition region in the outside image, for example. Thedisplay control unit may correct the image data for displaying thedisplay image in accordance with the state of the image superimpositionregion so that elements of the outside image are removed from thedisplay image to be observed by the observer, for example. In this case,the visibility of the display image can be increased, regardless of thestate of the outside image.

In the present technique, the display control unit may change manners ofdisplay of the display image when the image superimposition region isnot detected from the outside image, for example. The display isstopped, for example. Alternatively, the user is made to select asuperimposition position, and the display image is displayed in thatposition in a superimposed manner, for example. Alternatively, thedisplay image is displayed in a preset superimposition position in asuperimposed manner, for example. Alternatively, the display image isdisplayed in the superimposition position in which the display image ispreviously displayed, for example. Alternatively, the display positionis changed or the display is switched on and off in accordance with theduration of non-detection time, for example.

In the present technique, the display control unit may obtain thedisplay size and the display position for the control by performingtemporal smoothing on display sizes and display positions on the displaydevice, the display sizes and the display positions being determined bythe image superimposition region that is cyclically detected. In thiscase, even if there is a large change in the position or the size of theimage superimposition region cyclically detected for each frame, forexample, the display image can be stably superimposed and displayed onthe outside image.

In the present technique, the display control unit may also changemanners of display of the display image when a change in the outsideimage is detected, for example. The display is stopped, for example.Alternatively, the user is made to select a superimposition position,and the display image is displayed in that position in a superimposedmanner, for example. Alternatively, the display image is displayed in apreset superimposition position in a superimposed manner, for example.Alternatively, the display image is displayed in the superimpositionposition in which the display image is previously displayed, forexample. Alternatively, the display position is changed or the displayis switched on and off in accordance with the duration of changedetection, for example.

In the present technique, the optical system may include a first opticalsystem that superimposes a left-eye image displayed on a first displaydevice onto an outside image and leads the left-eye image to the lefteye of the observer, and a second optical system that superimposes aright-eye image displayed on a second display device onto the outsideimage and leads the right-eye image to the right eye of the observer,for example. The display control unit may control disparities of theleft-eye image and the right-eye image so that the depth position of thestereoscopic image to be perceived by the observer through the left-eyeimage and the right-eye image is located closer to the front side thanthe depth position of the region on which the stereoscopic image is tobe superimposed in the outside image, for example. In this case, thedisplay image (stereoscopic image) can be superimposed and displayed onthe outside image without causing any inconsistency in depth.

Another concept of the present technique lies in an image displayapparatus that includes:

an optical system that superimposes a display image displayed on adisplay device onto an outside image, and leads the display image to anobserver; and

a display control unit that has a first control mode for performingcontrol so that the display image is displayed in a region on which aline of sight of the observer concentrates in the outside image, and asecond control mode for performing control so that the display image isdisplayed in a region outside the region on which the line of sight ofthe observer concentrates in the outside image.

In the present technique, the image displayed on the display device issuperimposed on the outside image and is led to an eye of an observer bythe optical system. In a case where the optical system is an enlargingoptical system, the image displayed on the display device is opticallyenlarged, and is observed as an enlarged virtual image by the observer.

The display control unit controls display of the display image to besuperimposed and displayed on the outside image in the first controlmode or the second control mode. In the first control mode, control isperformed so that the display image is displayed in the region on whichthe line of sight of the observer concentrates in the outside image. Inthe second control mode, control is performed so that the display imageis displayed in a region outside the region on which the line of sightof the observer concentrates in the outside image.

As described above, in the present technique, display of the displayimage to be superimposed and displayed on the outside image can becontrolled in the first control mode or the second control mode. Thatis, the display image can be displayed in the region on which the lineof sight of the observer concentrates in the outside image, and thedisplay image can be displayed in a region outside the region on whichthe line of sight of the observer concentrates in the outside image, sothat the display of the display image does not obstruct any activity.

In the present technique, the display control unit may perform controlin the first control mode when the observer is not moving, and performcontrol in the second control mode when the observer is moving, forexample. In this case, when the observer is not moving, control isautomatically switched to the first control mode, and the display imageis displayed in the region on which the line of sight of the observerconcentrates in the outside image. That is, the observer does not needto switch modes to concentrate on the display image in this case, andaccordingly, higher user friendliness is achieved.

In the present technique, the display control unit may change manners ofdisplay of the image in accordance with the state of the region on whichthe display image is to be superimposed in the outside image, forexample. The display control unit may correct the image data fordisplaying the display image in accordance with the state of the regionso that elements of the outside image are removed from the display imageto be observed by the observer, for example. In this case, thevisibility of the display image can be increased, regardless of thestate of the outside image.

In the present technique, the optical system may include a first opticalsystem that superimposes a left-eye image displayed on a first displaydevice onto an outside image and leads the left-eye image to the lefteye of the observer, and a second optical system that superimposes aright-eye image displayed on a second display device onto the outsideimage and leads the right-eye image to the right eye of the observer,for example. The display control unit may control (adjust) thedisparities of the left-eye image and the right-eye image so that thedepth position of a stereoscopic image to be perceived by the observerthrough the left-eye image and the right-eye image is located closer tothe front side than the depth position of the region on which thestereoscopic image is to be superimposed in the outside image, forexample. In this case, the display image (stereoscopic image) can besuperimposed and displayed on the outside image without causing anyinconsistency in depth.

Yet another concept of the present technique lies in an image displayapparatus that includes:

an optical system that superimposes a display image displayed on adisplay device onto an outside image, and leads the display image to aneye of an observer; and

a display control unit that changes the display state of the displayimage in accordance with the state of the region on which the displayimage is to be superimposed in the outside image.

In the present technique, the image displayed on the display device issuperimposed on the outside image and is led to an eye of an observer bythe optical system. In a case where the optical system is an enlargingoptical system, the image displayed on the display device is opticallyenlarged, and is observed as an enlarged virtual image by the observer.

The display control unit changes the display state of the display imagein accordance with the state of the region on which this display imageis to be superimposed in the outside image. The state of the region inthe outside image is acquired based on captured image data obtained byforming the outside image, for example. The state of the region in theoutside image may be acquired in a cloud, for example. The displaycontrol unit may correct the image data for displaying the display imagein accordance with the state of the region so that elements of theoutside image are removed from the display image to be observed by theobserver, for example.

As described above, in the present technique, the display state of thedisplay image is changed in accordance with the state of the region onwhich this display image is to be superimposed in the outside image.Accordingly, components of the outside image can be removed from thedisplay image to be observed by the observer, and the visibility of thedisplay image can be increased, regardless of the state of the outsideimage.

Still another concept of the present technique lies in an image displayapparatus that includes:

a first optical system that superimposes a left-eye image displayed on afirst display device onto an outside image, and leads the left-eye imageto the left eye of an observer;

a second optical system that superimposes a right-eye image displayed ona second display device onto the outside image, and leads the right-eyeimage to the right eye of the observer; and

a display control unit that controls the disparities of the left-eyeimage and the right-eye image so that the depth position of thestereoscopic image to be perceived by the observer through the left-eyeimage and the right-eye image is located closer to the front side thanthe depth position of the region on which the stereoscopic image is tobe superimposed in the outside image.

In the present technique, the left-eye image displayed on the firstdisplay device is superimposed on the outside image and is led to theleft eye of the observer by the first optical system. Likewise, theright-eye image displayed on the second display device is superimposedon the outside image and is led to the right eye of the observer by thesecond optical system. In a case where the optical systems are enlargingoptical systems, the left-eye image and the right-eye image areoptically enlarged, and are observed as enlarged virtual images by theobserver.

The display control unit controls the disparities of the left-eye imageand the right-eye image. In this case, the disparities of the left-eyeimage and the right-eye image are controlled (adjusted) so that thedepth position of the stereoscopic image to be perceived by the observerthrough the left-eye image and the right-eye image is located closer tothe front side than the depth position of the region on which thestereoscopic image is to be superimposed in the outside image.

As described above, in the present technique, the disparities of theleft-eye image and the right-eye image are controlled based on the depthposition of the region in the outside image on which the display image(stereoscopic image) is to be superimposed, so that the depth positionof the display image (stereoscopic image) becomes closer to the frontside than the depth position of the region. Accordingly, the displayimage (stereoscopic image) can be superimposed and displayed on theoutside image without causing any inconsistency in depth.

Effects of the Invention

According to the present technique, a display image can be superimposedand displayed on an outside image in a preferred manner.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram showing a schematic example structure of anoptically-transmissive (binocular) head mount display as an embodiment.

FIGS. 2(a) to 2(f) are diagrams for explaining a disparity mapindicating the depth positions of the respective pixels in an outsideimage and a disparity map indicating the depth positions of therespective pixels in a display image (a stereoscopic image).

FIG. 3 is a diagram for explaining a temporal smoothing process for thedisplay size and the display position for display control on a displayimage.

FIG. 4 is a diagram for explaining a case where there is inconsistencyin the sense of depth with respect to the depth position of a displayimage (a stereoscopic image).

FIG. 5 is a diagram for explaining a case where there is noinconsistency in the sense of depth with respect to the depth positionof a display image (a stereoscopic image).

FIGS. 6(a) to 6(e) are diagrams for explaining an example process to beperformed to adjust the depth position of a display image (astereoscopic image) so as not to cause inconsistency in the sense ofdepth.

FIGS. 7(a) to 7(c) are diagrams for explaining an example process to beperformed to adjust the depth position of a display image (astereoscopic image) so as not to cause inconsistency in the sense ofdepth.

FIG. 8 is a diagram for explaining a case where a display image (astereoscopic image) is displayed in a region outside the region on whichthe line of sight of the observer concentrates in an outside image.

FIG. 9 is a diagram showing an example of superimposed display of adisplay image (a stereoscopic image) on a blue-sky portion in an outsideimage.

FIGS. 10(a) to 10(d) are diagrams for explaining a case where a displayimage is displayed in an image superimposition region such as a flatregion detected from an outside image.

FIG. 11 is a flowchart showing an example of procedures for performingdisplay control at the display control unit.

FIGS. 12(a) to 12(d) are diagrams schematically showing example layoutsof components of an optically-transmissive head mount display.

FIG. 13 is a diagram showing another example structure of anoptically-transmissive head mount display.

FIG. 14 is a diagram showing yet another example structure of anoptically-transmissive head mount display.

FIG. 15 is a diagram showing still another example structure of anoptically-transmissive head mount display.

MODES FOR CARRYING OUT THE INVENTION

The following is a mode for carrying out the invention (hereinafterreferred to as the “embodiment”). Explanation will be made in thefollowing order.

1. Embodiment

2. Modifications

<1. Embodiment>

[Example Structure of an Optically-Transmissive Head Mount Display]

FIG. 1 schematically shows an example structure of a head mount display(HMD) 100 of an optically-transmissive type as an embodiment. Thisexample structure is a binocular HMD. This HMD 100 includes a left glasslens unit 101L and a right glass lens unit 101R. The glass lens unit101L and the glass lens unit 101R are integrally connected by aconnecting member 102.

Each of the glass lens units 101L and 101R is formed by integrating aglass lens and a holographic optical element (HOE) sheet. This HOE sheethas a half-mirror-like function to combine outside light and displaylight, and a function of a concave surface or an adjustable surface toenlarge a display image.

Infrared sensors 103L and 103R are attached to the glass lens units 101Land 101R, respectively. The infrared sensor 103L is provided in thecenter position of the glass lens unit 101L in the horizontal direction(the center position of the left-eye optical system in the horizontaldirection), for example. The infrared sensor 103R is also provided inthe center position of the glass lens unit 101R in the horizontaldirection (the center position of the right-eye optical system in thehorizontal direction), for example.

Sensor outputs of the infrared sensors 103L and 103R are used forestimating eye positions by a scleral reflection method. The scleralreflection method is a method that utilizes a difference in reflectancebetween the cornea (the black part of the eye) and the sclera (the whitepart of the eye). In this case, an infrared sensor horizontally scansweak infrared rays emitted onto an eye of an observer, and detects thereflected light. Since there is a large difference between the intensityof the light reflected from the cornea (the black part of the eye) andthe intensity of the light reflected from the sclera (the white part ofthe eye), the position of the eye of the observer can be estimated froma sensor output.

A gyro sensor 104 is also attached to the glass lens unit 101L. A sensoroutput of the gyro sensor 104 is used for determining whether there is achange in the image of the outside, and whether the observer (user) ismoving. A sensor output of the gyro sensor 104 is also used fordetermining whether there is a change in the image of the outside beingobserved by the observer through the glass lens units 101L and 101R.

A camera 105L is also provided in the center position of the glass lensunit 101L in the horizontal direction (the center position of theleft-eye optical system in the horizontal direction). The camera 105Lcaptures an image (left-eye imagery) of the outside being observed withthe left eye of the observer through the glass lens unit 101L, andoutputs the captured image data. Likewise, a camera 105R is alsoprovided in the center position of the glass lens unit 101R in thehorizontal direction (the center position of the right-eye opticalsystem in the horizontal direction). The camera 105R captures an image(right-eye imagery) of the outside being observed with the right eye ofthe observer through the glass lens unit 101R, and outputs the capturedimage data.

Outputs of the cameras 105L and 105R are used for obtaining informationabout the depth position of an outside image on which a stereoscopicimage is to be superimposed and displayed. Outputs of the cameras 105Land 105R are also used for determining whether there is a change in theimage of the outside being observed by the observer through the glasslens units 101L and 101R. Outputs of the cameras 105L and 105R are usedfor obtaining information (such as luminance information and colorinformation) indicating the state of an outside image on which astereoscopic image is to be superimposed and displayed. Outputs of thecameras 105L and 105R are also used for detecting an imagesuperimposition region, or a flat region in this embodiment, from anoutside image.

The HMD 100 also includes display drivers 111L and 111R, and displays112L and 112R. Each of the displays 112L and 112R is formed with aliquid crystal display (LCD), for example. The display 112L is driven bythe display driver 111L based on left-eye image data, and displays aleft-eye image for making the observer perceive a stereoscopic image.The display 112R is driven by the display driver 111R based on right-eyeimage data, and displays a right-eye image for making the observerperceive a stereoscopic image.

The HMD 100 also includes an eye position estimating unit 121, aline-of-sight estimating unit 122, a depth/structure estimating unit123, a display control unit 124, and a to-be-displayed image generatingunit 125. The eye position estimating unit 121 estimates positions ofthe left eye and the right eye of the observer based on sensor outputsfrom the infrared sensors 103L and 103R. The line-of-sight estimatingunit 122 estimates a line of sight of the observer based on the resultof the left-eye and right-eye position estimation performed by the eyeposition estimating unit 121.

The depth/structure estimating unit 123 calculates a disparity mapindicating the depth position of each pixel in an outside image based oncaptured image data from the cameras 105L and 105R. FIG. 2(a) shows anexample of the left-eye imagery and the right-eye imagery of an outsideimage, and FIG. 2(b) shows an example of a disparity map correspondingto the left-eye imagery and the right-eye imagery. This example is animage formed by displaying the disparities of the respective pixels aspixel data, and brighter portions are shown in depth positions that arecloser to the front side. FIG. 2(c) shows an example of a disparityhistogram of the entire screen.

The depth/structure estimating unit 123 also calculates a disparity mapthat indicates the depth positions of the respective pixels in a displayimage (a stereoscopic image) based on left-eye and right-eye image dataserving as display image data. FIG. 2(d) shows an example of a left-eyeimage and a right-eye image, and FIG. 2(e) shows an example of adisparity map corresponding to the left-eye image and the right-eyeimage. This example is also an image formed by displaying thedisparities of the respective pixels as pixel data, and brighterportions are shown in depth positions that are closer to the front side.FIG. 2(f) shows an example of a disparity histogram of the entirescreen.

The depth/structure estimating unit 123 also detects an imagesuperimposition region from an outside image based on captured imagedata from the cameras 105L and 105R. The depth/structure estimating unit123 detects a region containing only low-frequency components in thehorizontal direction and the vertical direction (a flat region) as animage superimposition region, for example. This image superimpositionregion is of course a region in which a display image is to be displayedin a superimposed manner, and therefore, has a sufficient size in boththe horizontal direction and the vertical direction. In this case, notonly one image superimposition region but more than one imagesuperimposition region may be detected from an outside image.

The depth/structure estimating unit 123 also determines a display sizeand a display position of a display image based on the detected imagesuperimposition region. Performing the above described imagesuperimposition region detection cyclically or for each frame, forexample, the depth/structure estimating unit 123 also determines adisplay size and a display position of a display image for each frame.

So as to stabilize a display size and a display position of a displayimage, the depth/structure estimating unit 123 performs temporalsmoothing on the display sizes and display positions determined for therespective frames, and determines the display size and the displayposition for display control. FIG. 3 shows an example of a temporalsmoothing process for a display position. Coordinate filtering (such asaveraging, IIR, majority operation, or the like) is performed to achievetemporal stabilization. In the case of filtering for averaging, forexample, the coordinates (x, y) of the display region that is actuallyused are determined based on the mathematical formula (1) shown below.

[Mathematical Formula 1]

$\begin{matrix}{{\left( {x,y} \right) = {{\left( {{\frac{1}{N}{\sum x_{i}}},{\frac{1}{N}{\sum y_{i}}}} \right)\mspace{14mu} i} = 0}},\ldots\mspace{14mu},{N - 1}} & (1)\end{matrix}$

The depth/structure estimating unit 123 also determines a depth positionof a display image (a stereoscopic image) based on the display size andthe display position for display control that are determined in theabove described manner. In this case, the depth/structure estimatingunit 123 determines the depth position of a display image so that thedepth position of the display image is located closer to the front sidethan the depth position of the region in which the display image is tobe displayed in a superimposed manner in the outside image.

A depth position is determined as described above, so as to avoid anyinconsistency in the sense of depth when a display image (a stereoscopicimage) is superimposed and displayed on an image of the outside, whichis the real world. FIG. 4 schematically shows a case where aninconsistency is caused. If superimposed display is performed in a casewhere the depth position of an object A in an outside image is closer tothe front side while the depth position of an object B in a displayimage is closer to the back side, the object A is divided by the objectB, giving the observer an unnatural sense of depth. Such aninconsistence in the sense of depth makes the observer feel tired, forexample.

FIG. 5 schematically shows that an inconsistency in the sense of depthis avoided by placing a display image (a stereoscopic image) in a depthposition that is closer to the front side than the region in which thedisplay image is to be displayed in a superimposed manner in the outsideimage. As the depth position of the object B is made closer to the frontside than the depth position of the object A in the case where the depthposition of the object A in the outside image is closer to the frontside while the depth position of the object B in the display image iscloser to the back side, the observer can have a natural sense of deptheven if the object A is divided by the object B. As the sense of depthis adjusted in this manner, the observer can comfortably observe theobjects without feeling tired.

Specifically, the depth/structure estimating unit 123 determinesdisparities to be given to the left-eye image and the right-eye image.The depth/structure estimating unit 123 determines Ha, which is theaverage of disparities in the region in which a display image having adisplay size and a display position determined as described above isdisplayed in a superimposed manner in the outside image. Thedepth/structure estimating unit 123 also determines Hb, which is theaverage of disparities in the entire display image having the displaysize determined as described above. It should be noted that Hb can beobtained by calculating the average (see FIG. 2(f)) of the disparitiesin the display image prior to the above described size adjustment by n(n being the magnification of the display size).

For example, a case where a display image is superimposed on an outsideimage and is displayed as shown in FIG. 6(a) is now described. In thiscase, the disparities in the regions (indicated by rectangular frames)in which the left-eye image and the right-eye image of the display imageare displayed in a superimposed manner in a disparity map of theleft-eye imagery and the right-eye imagery of the outside image are usedas shown in FIG. 6(b), and the average of the disparities is Ha as shownin FIG. 6(c).

Each of the rectangular frames is drawn with two lines in FIG. 6(b),with the inner line matching the display size and the display positiondetermined as described above, the outer line being determined by addinga margin to the inner line. The outer rectangular frames are used asregions for obtaining Ha. This is because disparity adjustment might beperformed by changing the relative positions of the left-eye image andthe right-eye image as will be described later, as the depth position ofthe display image (the stereoscopic image) is made closer to the frontside than the depth position of the outside image.

FIG. 6(d) shows a disparity histogram (the same as FIG. 2(f)) of theentire display image prior to the size adjustment. FIG. 6(e) shows adisparity histogram of the entire display image after the display sizeis divided by n, and the average Hb of the disparities is n times largerthan the disparities in the entire display screen prior to the sizeadjustment.

The depth/structure estimating unit 123 compares the disparity averageHa related to the outside image that is determined in the abovedescribed manner, with the disparity average Hb related to the displayimage. The depth/structure estimating unit 123 then determines whetherthe depth position of the display image (the stereoscopic image) islocated closer to the front side than the depth position of thecorresponding region in the outside image by a certain distance or more,or whether the disparity average difference of the disparity average Hbwith respect to the disparity average Ha is H0 or larger, whichsatisfies the above condition.

When the disparity average difference is smaller than H0, thedepth/structure estimating unit 123 adjusts one or both of the displaypositions of the left-eye image and the right-eye image in thehorizontal direction, so as to make the disparity average Hb have adisparity average difference equal to or larger than H0. FIG. 7(a) showsan example of the disparity averages Ha and Hb prior to display positionadjustment, and shows a case where the disparity average difference issmaller than H0. In this case, one or both of the display positions ofthe left-eye image and the right-eye image in the horizontal directionare adjusted so that the disparity average difference is made equal toor larger than H0 as shown in FIG. 7(b).

In the above description, one or both of the display positions of theleft-eye image and the right-eye image in the horizontal direction areadjusted so that the disparity average difference of the disparityaverage Hb with respect to the disparity average Ha becomes equal to orlarger than H0. Instead, as shown in FIG. 7(c), one or both of thedisplay positions of the left-eye image and the right-eye image in thehorizontal direction may be adjusted so that a difference between 90% ofthe disparities in the disparity histogram of the outside image and 10%of the disparities in the disparity histogram of the display imagebecomes equal to or larger than a predetermined threshold value, forexample.

The display control unit 124 controls display of the display image basedon the result of the line-of-sight estimation performed by theline-of-sight estimating unit 122, a sensor output of the gyro sensor104, information about the display size and the display position of thedisplay image determined by the depth/structure estimating unit 123, andthe like. Although not shown in the drawings, a user operation signal isalso supplied to the display control unit 124.

In a case where an instruction to display the display image is issuedthrough a user operation, the display control unit 124 basicallycontrols display of the display image so that the display image isdisplayed in the display size and the display position determined by thedepth/structure estimating unit 123.

When any flat region that is an image superimposition region is notdetected by the depth/structure estimating unit 123, or when anyinformation about a display size and a display position is not suppliedfrom the depth/structure estimating unit 123, the display control unit124 changes display conditions.

The display control unit 124 performs control so that display of thedisplay image is stopped, for example. Alternatively, the displaycondition control unit 124 performs control to make the user select asuperimposition position so that the display image is displayed in thatposition, for example. Alternatively, the display condition control unit124 performs control so that the display image is displayed in a presetsuperimposition position, for example. Alternatively, the displaycondition control unit 124 performs control so that the display image isdisplayed in the previously displayed superimposition position, forexample.

The display control unit 124 also controls the display position of thedisplay image or switching on and off of the display of the displayimage in accordance with the duration of non-detection time. In thiscase, control is performed so that the display image is displayed in theprevious display position until the duration reaches a first point oftime, control is performed so that the display image is displayed in apreset position until the duration reaches a second point of time afterreaching past the first point of time, and control is performed so thatthe display is stopped when the duration is past the second point oftime.

The display control unit 124 also changes display conditions when acheck is made to determine whether there is a change in the outsideimage and a change is detected based on the gyro sensor 104. The displaycondition control unit 124 performs control so that display of thedisplay image is stopped, for example. Alternatively, the displaycondition control unit 124 performs control to make the user select asuperimposition position so that the display image is displayed in thatposition, for example. Alternatively, the display condition control unit124 performs control so that the display image is displayed in a presetsuperimposition position, for example. Alternatively, the displaycondition control unit 124 performs control so that the display image isdisplayed in the previously displayed superimposition position, forexample.

The display control unit 124 also controls the display position of thedisplay image or switching on and off of the display of the displayimage in accordance with the duration of change. In this case, controlis performed so that the display image is displayed in the previousdisplay position until the duration reaches a first point of time,control is performed so that the display image is displayed in a presetposition until the duration reaches a second point of time afterreaching past the first point of time, and control is performed so thatthe display is stopped when the duration is past the second point oftime.

The display control unit 124 also changes display conditions based on aline-of-sight estimation result from the line-of-sight estimating unit122. In accordance with a mode that is set by the user (the observer),the display condition control unit 124 performs control in the mannerdescribed below. The user can set “automatic control mode”, “firstcontrol mode”, or “second control mode”.

Where “first control mode” is set, the display control unit 124 performscontrol so that the display image is displayed in the region on whichthe line of sight concentrates or the region that matches the line ofsight. Where “second control mode” is set, the display control unit 124performs control so that the display image is displayed in a regionoutside the region on which the line of sight concentrates or a regionoutside the region that matches the line of sight.

Where “automatic control mode” is set, the display control unit 124performs control in the manner described below depending on whether theuser (observer) is moving. Specifically, when the user is not moving,the display control unit 124 performs control so that the display imageis displayed in the region on which the line of sight concentrates as ina case where “first control mode” is set. When the user is moving, thedisplay control unit 124 performs control so that the display image isdisplayed in a region outside the region on which the line of sightconcentrates as in a case where “second control mode” is set. Thedisplay control unit 124 determines whether the observer is moving basedon a sensor output of the gyro sensor 104.

Under the control of the display control unit 124, the to-be-displayedimage generating unit 125 generates left-eye and right-eye image datafor display so that the display image is displayed in the display sizeand the display position determined by the depth/structure estimatingunit 123 at the time of display of the display image. In this case, areduction process and a moving process (a geometric transformationprocess) are performed on left-eye and right-eye image data suppliedfrom outside, to obtain the left-eye and right-eye image data fordisplay. Under the control of the display control unit 124, theto-be-displayed image generating unit 125 also corrects the left-eye andright-eye image data for display so that the display state of thedisplay image is changed in accordance with the state of the region onwhich the display image is to be displayed in the outside image. In thiscase, the correction is performed so that elements (components) of theoutside image are removed from the display image to be observed by theobserver.

An example of image data correction to be performed by theto-be-displayed image generating unit 125 is now described. The outsideimage is represented by Ireal, and the display image is represented byIdisp. Each of Idisp and Ireal is divided into blocks each consisting of(N×N) pixels. The pixel at the coordinates (i, j) in the outside imageIreal is represented by Ireal (i, j), and the pixel at the coordinates(i, j) in the display image Idisp is represented by Idisp (i, j).

As for the data of each pixel in a block of coordinates (s, t), theto-be-displayed image generating unit 125 performs correction asindicated in the mathematical formula (2) shown below. Here, arepresents the correction coefficient, and the user (observer) canadjust the value of the correction coefficient as necessary. Also,clip(x) represents a function to perform a saturation calculation on xinto a certain value range (0 to 255, for example). Although notdescribed in detail, this pixel data correction is performed on data ofthe respective colors of red, green, and blue.

[Mathematical Formula 2]

$\begin{matrix}{{I_{disp}^{\prime}\left( {i,j} \right)} = {{clip}\left( {{I_{disp}\left( {i,j} \right)} - {\alpha\frac{1}{N^{2}}{\sum\limits_{{({i,j})} \in {{block}{({s,t})}}}{I_{real}\left( {i,j} \right)}}}} \right)}} & (2)\end{matrix}$

As for the second term in the parenthesis, a value obtained by smoothinga correction value of a surrounding block may be used as indicated inthe mathematical formula (3) shown below.

[Mathematical Formula 3]

$\begin{matrix}{\frac{1}{9\; N^{2}}{\sum\limits_{{- 1} \leq u \leq {1 - 1} \leq v \leq 1}{\sum\limits_{{({i,j})} \in {{block}{({{s + u},{t + v}})}}}{I_{real}\left( {i,j} \right)}}}} & (3)\end{matrix}$

Operation of the HMD 100 shown in FIG. 1 is now described. Left-eyeimage data generated by the to-be-displayed image generating unit 125 issupplied to the display driver 111L. The display 112L is driven by thisdisplay driver 111L, and the left-eye image is displayed on this display112L. Meanwhile, right-eye image data generated by the to-be-displayedimage generating unit 125 is supplied to the display driver 111R. Thedisplay 112R is driven by this display driver 111R, and the right-eyeimage is displayed on this display 112R.

The light from the left-eye image displayed on the display 112L issuperimposed on the outside image at the glass lens unit 101L, andreaches the left eye of the observer. As a result, the left-eye imagesuperimposed on the outside image (left-eye imagery) is observed withthe left eye of the observer. Likewise, the light from the right-eyeimage displayed on the display 112R is superimposed on the outside imageat the glass lens unit 101R, and reaches the right eye of the observer.As a result, the right-eye image superimposed on the outside image(right-eye imagery) is observed with the right eye of the observer. Asthe left-eye image and the right-eye image superimposed on the outsideimage are observed with the left eye and the right eye of the observer,respectively, the display image superimposed and displayed on theoutside image is perceived as a stereoscopic (3D) image by the observer.

A sensor output of the infrared sensor 103L provided in the centerposition of the glass lens unit 101L in the horizontal direction (thecenter position of the left-eye optical system in the horizontaldirection) is supplied to the eye position estimating unit 121.Likewise, a sensor output of the infrared sensor 103R provided in thecenter position of the glass lens 101R in the horizontal direction (thecenter position of the right-eye optical system in the horizontaldirection) is supplied to the eye position estimating unit 121.

The eye position estimating unit 121 estimates positions of the left eyeand the right eye of the observer based on the sensor outputs from theinfrared sensors 103L and 103R. The line-of-sight estimating unit 122then estimates a line of sight of the observer based on the result ofthe left-eye and right-eye position estimation performed by the eyeposition estimating unit 121. The result of this line-of-sightestimation is supplied to the display control unit 124.

A sensor output of the gyro sensor 104 attached to the glass lens unit101L is supplied to the display control unit 124. An output (capturedleft-eye image data) of the camera 105L provided in the center positionof the glass lens unit 101L in the horizontal direction (the centerposition of the left-eye optical system in the horizontal direction) issupplied to the depth/structure estimating unit 123.

Likewise, an output (captured right-eye image data) of the camera 105Rprovided in the center position of the glass lens unit 101R in thehorizontal direction (the center position of the right-eye opticalsystem in the horizontal direction) is supplied to the depth/structureestimating unit 123. The left-eye and right-eye image data, which is thedisplay image data, is further supplied to the depth/structureestimating unit 123.

The depth/structure estimating unit 123 detects a flat region as animage superimposition region from the outside image based on thecaptured image data from the cameras 105L and 105R. The depth/structureestimating unit 123 then determines a display size and a displayposition of the display image based on the detected flat region. In thiscase, temporal smoothing is performed on display sizes and displaypositions determined for the respective frames, and the display size andthe display position of the display image are stabilized.

The depth/structure estimating unit 123 also calculates a disparity mapindicating the depth positions of the respective pixels in the outsideimage based on the captured image data from the cameras 105L and 105R,and calculates a disparity map indicating the depth positions of therespective pixels in the display image (stereoscopic image) based on theleft-eye and right-eye image data, which is the display image data. Thedepth/structure estimating unit 123 then determines the depth positionof the display image (stereoscopic image) based on the display size andthe display position for display control that are determined in theabove described manner, and the disparity maps calculated in the abovedescribed manner. In this case, the depth position of the display imageis determined so that the display image is located closer to the frontside than the depth position of the region in which the display image isto be displayed in a superimposed manner in the outside image.

The depth/structure estimating unit 123 determines whether the depthposition of the display image (stereoscopic image) satisfies thecondition that the display image is located at a certain distance orlonger on the front side from the depth position of the regioncorresponding to the outside image. If the condition is not satisfied,the depth/structure estimating unit 123 performs disparity adjustment toadjust one or both of the display positions of the left-eye image andthe right-eye image in the horizontal direction so that the condition issatisfied. The information about the display size and the displayposition for display control that are determined by the depth/structureestimating unit 123 in this manner is supplied to the display controlunit 124.

The display control unit 124 controls display of the display image basedon the result of the line-of-sight estimation performed by theline-of-sight estimating unit 122, the sensor output of the gyro sensor104, and the information about the display size and the display positiondetermined by the depth/structure estimating unit 123. In this case, thedisplay control unit 124 basically performs control so that the displayimage is displayed in the display size and the display positiondetermined by the depth/structure estimating unit 123 at the time ofdisplay of the display image.

When a flat region as an image superimposition region is not detected bythe depth/structure estimating unit 123, the display control unit 124changes manners of display. When a change in the outside image isdetected based on the gyro sensor 104, the display control unit 124 alsochanges manners of display.

The display control unit 124 also changes manners of display based onthe line-of-sight estimation result from the line-of-sight estimatingunit 122. This control is performed in accordance with a mode that isset by the user (observer). The user can set “automatic control mode”,“first control mode”, or “second control mode”, for example.

Where “first control mode” is set, control is performed so that thedisplay image is displayed in the region on which the line of sightconcentrates. Where “second control mode” is set, control is performedso that the display image is displayed in a region outside the region onwhich the line of sight concentrates. Where “automatic mode” is set,control is performed so that the display image is displayed outside theregion on which the line of sight concentrates when the observer ismoving, and control is performed so that the display image is displayedin the region on which the line of sight concentrates when the observeris not moving.

As described above, where “second control mode” is set, the displayimage (stereoscopic image) can be displayed in a region outside theregion on which the line of sight of the observer concentrates in theoutside image, and the display image can be displayed in such a manneras not to obstruct any activity. In this case, the observer views thedisplay image while doing some other thing. FIG. 8 shows an example ofsuch display. In this example of display, the line of sight of theobserver concentrates on the stove in the kitchen, and therefore, thedisplay image is displayed in a wall portion that is away from the stoveportion.

The left-eye and right-eye image data, which is the display image data,is supplied to the to-be-displayed image generating unit 125. Thecaptured image data from the cameras 105L and 105R is also supplied tothis to-be-displayed image generating unit 125. Under the control of thedisplay control unit 124, the to-be-displayed image generating unit 125generates the left-eye and right-eye image data for displaying thedisplay image so that the display image is displayed in the determineddisplay size and the determined display position.

In this case, a reduction process and a moving process are performed onleft-eye and right-eye image data supplied from outside, to obtain theleft-eye and right-eye image data for display. In this case, the displaysize and the display position are electronically changed. When thedisplay of the display image is stopped, the generation of the left-eyeand right-eye image data is stopped.

The to-be-displayed image generating unit 125 also corrects the left-eyeand right-eye image data so that the display state of the display imageis changed in accordance with the state of the region on which thedisplay image is to be displayed in the outside image (see themathematical formula (2)). As the image data is corrected in thismanner, the visibility of the display image can be increased, regardlessof the state of the outside image.

FIG. 9 shows an example of superimposed display of a display image (astereoscopic image) on a blue-sky portion in an outside image. If theimage data has not been corrected in such a case, the display imageappears bluish to the observer due to influence of the blue sky.However, as the image data is corrected in this embodiment, theinfluence of the blue sky is reduced, and the observer can view thedisplay image in a preferred state.

The left-eye image data for display generated by the to-be-displayedimage generating unit 125 is supplied to the display driver 111L, andthe left-eye image corresponding to this left-eye image data isdisplayed on the display 112L. The right-eye image data for displaygenerated by the to-be-displayed image generating unit 125 is suppliedto the display driver 111R, and the right-eye image corresponding tothis right-eye image data is displayed on the display 112R.

As a result, the left-eye image and the right-eye image superimposed onthe outside image are observed with the left eye and the right eye ofthe observer, respectively, and the display image (stereoscopic image)superimposed and displayed in an appropriate position and an appropriatesize on the outside image is perceived in a depth position in front ofthe outside image by the observer.

In this case, the display image is basically displayed in an imagesuperimposition region such as a flat region detected from the outsideimage, and accordingly, it becomes easier for the observer to visuallyrecognize the display image superimposed and displayed on the outsideimage. An example case where the outside image is the one shown in FIG.10(a) and the display image is the one shown in FIG. 10(b) is nowdescribed.

In this case, the flat region in the wall portion shown on the uppermiddle side in FIG. 10(a) is detected as an image superimpositionregion. The display size and the display position of the display imageshown in FIG. 10(b) are adjusted as shown in FIG. 10(c) so that thedisplay image is displayed in this flat region. The display image isthen superimposed and displayed on the outside image as shown in FIG.10(d).

The flowchart in FIG. 11 shows an example of procedures for performingdisplay control at the display control unit 124 of the HMD 100 shown inFIG. 1. In step ST1, the display control unit 124 starts operation whena power-on operation is performed by a user, for example. In step ST2,the display control unit 124 initializes the display position of thedisplay image to a preset value, for example. A display size is uniquelydetermined for the display position initialized in this manner.

In step ST3, the display control unit 124 determines whether imagedisplay is to be performed. If an image display setting operation isperformed by the user, for example, the display control unit 124determines that image display is to be performed. When image display isto be performed, the display control unit 124 in step ST4 determines thevalue of the mode set by the user.

When the value of the set mode indicates “second control mode”, thedisplay control unit 124 in step ST5 selects a region outside the lineof sight in the outside image as the object to be displayed. When thevalue of the set mode indicates “first control mode”, the displaycontrol unit 124 in step ST6 selects the region that matches the line ofsight in the outside image as the object to be displayed.

When the value of the set mode indicates “automatic mode”, the displaycontrol unit 124 in step ST7 determines whether the user is currentlymoving. If the user is currently moving, the display control unit 124 instep ST5 selects a region outside the line of sight in the outside imageas the object to be displayed. If the user is currently not moving, thedisplay control unit 124 in step ST6 selects a region that matches theline of sight in the outside image as the object to be displayed.

After carrying out the procedure of step ST5 or ST6, the display controlunit 124 moves on to the procedure of step ST8. In step ST8, the displaycontrol unit 124 selects a flat region in the object to be displayed asthe display position. When there is more than one flat region in theobject to be displayed, the region with the largest area is selected asthe display position. In this manner, the display size and the displayposition of the display image are determined.

In step ST9, the display control unit 124 determines whether there is achange in the outside image. If there is a change in the outside image,the duration of the change is determined in step ST10. If the durationis shorter than “th1”, the display control unit 124 in step ST11 doesnot change the display position. If the duration is equal to or longerthan “th1” but is shorter than “th2”, the display control unit 124 instep ST12 changes the display position to the preset position. If theduration is equal to or longer than “th2”, the display control unit 124in step ST13 stops the display of the display image.

After carrying out the procedure of step ST13, the display control unit124 returns to the procedure of step ST2, and repeats the sameprocedures as those described above. After carrying out the procedure ofstep ST11 or ST12, the display control unit 124 moves on to theprocedure of step ST14. In step ST14, the display control unit 124performs temporal smoothing on the display positions (display sizes). Inthis manner, even a rapid change of the display position (display size)can be changed to a smooth change.

In step ST15, the display control unit 124 adjusts the depth position ofthe display image (stereoscopic image) in accordance with the depthposition of the outside image. As described above, when the displayimage is to be displayed in a flat region in the outside image, thedepth position of the display image is adjusted by generating theleft-eye and right-eye image data for display in accordance with thedisplay size and the display position provided from the depth/structureestimating unit 123. In a case where the display image is to bedisplayed in the preset position, the display positions of the left-eyeand right-eye display images are moved and adjusted in the horizontaldirection in accordance with the depth position of the outside image inthe preset position.

In step ST16, the display control unit 124 corrects the left-eye andright-eye image data for display in accordance with the state of thedisplay region in the outside image. In step ST17, the display controlunit 124 performs image display. Specifically, the left-eye andright-eye image data for display is supplied from the to-be-displayedimage generating unit 125 to the display drivers 111L and 111R,respectively, and the left-eye image and the right-eye image aredisplayed on the displays 112L and 112R.

In step ST18, the display control unit 124 determines whether the imagehas come to an end. If an image display canceling operation is performedby the user, for example, the display control unit 124 determines thatthe image has come to an end. If the image has come to an end, thedisplay control unit 124 in step ST19 ends the display control process.If the image has not come to an end, the display control unit 124returns to step ST3, and repeats the same procedures as those describedabove.

For ease of explanation, the above described example of the proceduresshown in the flowchart in FIG. 11 is based on the assumption that a flatregion always exists in the object to be displayed. However, there mightbe a case where any flat region does not exist in the region selected asthe object to be displayed in step ST5 or ST6. In such a case, thedisplay control unit 124 controls the display position of the displayimage or controls switching on and off of the display in accordance withthe duration of flat region non-detection time as described above, forexample.

Not all the components of the HMD 100 shown in FIG. 1 need to beincluded in the HMD main frame. Instead, some of the components may beplaced in a control box that is connected to the HMD in a wired orwireless manner, or some of the components may be placed in a cloudconnected to the HMD via a network.

FIGS. 12(a) to 12(d) schematically show example layouts of therespective components of the HMD 100. FIG. 12(a) shows a case where allthe components of the HMD 100 are placed in the HMD main frame. FIG.12(b) shows a case where some of the components of the HMD 100 areplaced in a control box. In this case, the eye position estimating unit121, the line-of-sight estimating unit 122, the depth/structureestimating unit 123, the display control unit 124, and theto-be-displayed image generating unit 125 are placed in the control box,and the rest of the components are placed in the HMD main frame, forexample.

FIG. 12(c) shows a case where some of the components of the HMD 100 areplaced in a cloud. In this case, the eye position estimating unit 121,the line-of-sight estimating unit 122, the depth/structure estimatingunit 123, the display control unit 124, and the to-be-displayed imagegenerating unit 125 are placed in the cloud, and the rest of thecomponents are placed in the HMD main frame, for example.

FIG. 12(d) shows a case where some of the components of the HMD 100 areplaced in a control box and a cloud. In this case, the eye positionestimating unit 121, the line-of-sight estimating unit 122, and theto-be-displayed image generating unit 125 are placed in the control box,the depth/structure estimating unit 123 and the display control unit 124are placed in the cloud, and the rest of the components are placed inthe HMD main frame, for example.

As described above, in the HMD 100 shown in FIG. 1, display control isperformed so that a display image is displayed in an imagesuperimposition region such as a flat region detected from an outsideimage, and accordingly, it becomes easier for the observer (user) tovisually recognize the display image superimposed and displayed on theoutside image.

Also, in the HMD 100 shown in FIG. 1, display of the display image to besuperimposed and displayed on the outside image can be controlled in“first control mode” or “second control mode”. Specifically, the displayimage can be displayed in the region on which the line of sight of theobserver (user) concentrates in the outside image, or the display imagecan be displayed in a region outside the region on which the line ofsight of the observer concentrates in the outside image, so that thedisplay of the display image does not obstruct any activity.

Also, in the HMD 100 shown in FIG. 1, “automatic mode” is set, so thatcontrol is performed in “first control mode” when the observer (user) isnot moving, and control is performed in “second control mode” when theobserver (user) is moving. Therefore, when the observer is not moving,control is automatically switched to “first control mode”, and thedisplay image is displayed in the region on which the line of sight ofthe observer concentrates in the outside image. That is, the observerdoes not need to switch modes to concentrate on the display image inthis case, and accordingly, higher user friendliness is achieved.

Also, in the HMD 100 shown in FIG. 1, the display state of a displayimage is changed in accordance with the state of the region on whichthis display image is to be superimposed in the outside image.Accordingly, components of the outside image can be removed from thedisplay image to be observed by the observer (user), and the visibilityof the display image can be increased, regardless of the state of theoutside image.

Also, in the HMD 100 shown in FIG. 1, the disparities of the left-eyeimage and the right-eye image are controlled based on the depth positionof the region in the outside image on which the display image(stereoscopic image) is to be superimposed, so that the depth positionof the display image (stereoscopic image) becomes closer to the frontside than the depth position of the region. Accordingly, the displayimage (stereoscopic image) can be superimposed and displayed on theoutside image without causing any inconsistency in depth.

<2. Modifications>

In the above described embodiment, the depth/structure estimating unit123 calculates a disparity map indicating the depth position of eachpixel in an outside image based on captured image data from the cameras105L and 105R. However, as shown in FIG. 13, it is possible to forma HMD100A in which a distance measuring sensor 106 is provided at theconnecting member 102, and the depth/structure estimating unit 123calculates a disparity map indicating the depth positions of therespective pixels in the outside image based on a sensor output of thedistance measuring sensor 106.

Also, in the above described embodiment, the left-eye and right-eyeimage data for display differs from the captured left-eye and right-eyeimage data of the outside image obtained by the cameras 105L and 105R.However, as shown in FIG. 14, it is possible to form a HMD 100B thatuses captured left-eye and right-eye image data as the left-eye andright-eye image data for display.

Although an example of a binocular HMD has been described in the aboveembodiment, the present technique can also be applied to a monocularHMD. FIG. 15 shows an example structure of a monocular HMD 100C. In FIG.15, the same components as those shown in FIG. 1 are denoted by the samereference numerals as those used in FIG. 1, and explanation of them isnot repeated herein.

Being a monocular HMD, this HMD 100C has one glass lens unit 101, whilethe HMD 100 shown in FIG. 1 has the two glass lens units 101L and 101R.An infrared sensor 103 that functions in the same manner as the infraredsensors 103L and 103R in the HMD 100 shown in FIG. 1 is provided in thecenter position of the glass lens unit 101 in the horizontal direction(the center position of the optical system in the horizontal direction).A sensor output of this infrared sensor 103 is sent to an eye positionestimating unit 114.

A gyro sensor 104 is also attached to the glass lens unit 101. A sensoroutput of the gyro sensor 104 is used for determining whether there is achange in the image of the outside, and whether the observer (user) ismoving. A sensor output of this gyro sensor 104 is sent to a displaycontrol unit 124.

A camera 105 is also provided in the center position of the glass lensunit 101 in the horizontal direction (the center position of the opticalsystem in the horizontal direction). The camera 105 functions in thesame manner as the cameras 105L and 105R in the HMD 100 shown in FIG. 1,captures the outside image being observed with the left eye or the righteye of the observer through the glass lens unit 101, and outputscaptured image data. This captured image data is sent to a structureestimating unit 123C.

The eye position estimating unit 121 estimates a position of an eye (theleft eye or the right eye) of the observer based on the sensor outputfrom the infrared sensor 103. The line-of-sight estimating unit 122estimates a line of sight of the observer based on the result of the eyeposition estimation performed by the eye position estimating unit 121.The result of this line-of-sight estimation is supplied to the displaycontrol unit 124.

The structure estimating unit 123C detects a flat region as an imagesuperimposition region from the outside image based on the capturedimage data from the camera 105. The structure estimating unit 123 thendetermines a display size and a display position of the display imagebased on the detected flat region. In this case, temporal smoothing isperformed on display sizes and display positions determined for therespective frames, and the display size and the display position of thedisplay image are stabilized. The information about the display size andthe display position for display control that are determined by thestructure estimating unit 123C in this manner is supplied to the displaycontrol unit 124.

The display control unit 124 controls display of the display image basedon the result of the line-of-sight estimation performed by theline-of-sight estimating unit 122, the sensor output of the gyro sensor104, and the information about the display size and the display positiondetermined by the structure estimating unit 123C. In this case, thedisplay control unit 124 basically performs control so that the displayimage is displayed in the display size and the display positiondetermined by the structure estimating unit 123C at the time of displayof the display image.

When a flat region as an image superimposition region is not detected bythe structure estimating unit 123C, the display control unit 124 changesmanners of display. When a change in the outside image is detected basedon the gyro sensor 104, the display control unit 124 also changesmanners of display.

The display control unit 124 also changes manners of display based onthe line-of-sight estimation result from the line-of-sight estimatingunit 122. This control is performed in accordance with a mode that isset by the user (observer). The user can set “automatic control mode”,“first control mode”, or “second control mode”, for example.

Where “first control mode” is set, control is performed so that thedisplay image is displayed in the region on which the line of sightconcentrates. Where “second control mode” is set, control is performedso that the display image is displayed in a region outside the region onwhich the line of sight concentrates. Where “automatic mode” is set,control is performed so that the display image is displayed outside theregion on which the line of sight concentrates when the observer ismoving, and control is performed so that the display image is displayedin the region on which the line of sight concentrates when the observeris not moving.

Image data is supplied to the to-be-displayed image generating unit 125.The captured image data from the camera 105 is also supplied to thisto-be-displayed image generating unit 125. Under the control of thedisplay control unit 124, the to-be-displayed image generating unit 125generates the image data for displaying the display image so that thedisplay image is displayed in the determined display size and thedetermined display position.

In this case, a reduction process and a moving process are performed onimage data supplied from outside, to generate the image data fordisplay. When the display of the display image is stopped, thegeneration of the image data is stopped. The to-be-displayed imagegenerating unit 125 also corrects the image data for display so that thedisplay state of the display image is changed in accordance with thestate of the region on which the display image is to be displayed in theoutside image (see the mathematical formula (2)).

Although the other aspects of the HMD 100C shown in FIG. 15 are notdescribed herein, the HMD 100C has the same design as the HMD 100 shownin FIG. 1, operates in the same manner as the HMD 100, and can achievethe same effects as those of the HMD 100.

In the above described embodiment, eye positions and a line of sight areestimated by using sensor outputs of the infrared sensors. However, astructure for estimating a line of sight of an observer (a user) is notlimited to that structure. For example, it is also possible to use anEOG (Electro-Oculogram) method, a face recognition technology, or thelike.

In the above described embodiment, the present technique is applied toan optically-transmissive head mount display. However, the presenttechnique is not limited to applications to optically-transmissive headmount displays, but can also be applied to other transmissive displayapparatuses. In this case, displaying a virtual image is not necessary.

The present technique may also be embodied in the structures describedbelow.

(1) An image display apparatus including:

an optical system that superimposes a display image displayed on adisplay device onto an outside image, and leads the display image to aneye of an observer; and

a display control unit that controls a display size and a displayposition of the display image on the display device so that the displayimage is displayed in an image superimposition region detected from theoutside image.

(2) The image display apparatus of (1), wherein the imagesuperimposition region is detected based on captured image data obtainedby forming the outside image.

(3) The image display apparatus of (1) or (2), wherein the displaycontrol unit controls the display size and the display position of thedisplay image by processing image data for displaying the display imageon the display device based on information about the imagesuperimposition region.

(4) The image display apparatus of any of (1) through (3), wherein thedisplay control unit changes a display state of the display image inaccordance with a state of the image superimposition region in theoutside image.

(5) The image display apparatus of (4), wherein the display control unitcorrects the image data for displaying the display image in accordancewith the state of the image superimposition region so that elements ofthe outside image are removed from the display image to be observed bythe observer.

(6) The image display apparatus of any of (1) through (5), wherein, whenthe image superimposition region is not detected from the outside image,the display control unit changes manners of display of the displayimage.

(7) The image display apparatus of any of (1) through (6), wherein thedisplay control unit obtains the display size and the display positionfor the control by performing temporal smoothing on display sizes anddisplay positions on the display device, the display sizes and thedisplay positions being determined by the image superimposition regionthat is cyclically detected.

(8) The image display apparatus of any of (1) through (7), wherein, whena change in the outside image is detected, the display control unitchanges manners of display of the display image.

(9) The image display apparatus of any of (1) through (8), wherein

the optical system includes a first optical system that superimposes aleft-eye image displayed on a first display device onto an outside imageand leads the left-eye image to the left eye of the observer, and asecond optical system that superimposes a right-eye image displayed on asecond display device onto the outside image and leads the right-eyeimage to the right eye of the observer, and

the display control unit controls disparities of the left-eye image andthe right-eye image so that a depth position of a stereoscopic image tobe perceived by the observer through the left-eye image and theright-eye image is located closer to the front side than a depthposition of a region in which the stereoscopic image is displayed in asuperimposed manner in the outside image.

(10) An image display method including the steps of:

superimposing a display image displayed on a display device onto anoutside image, and leading the display image to an eye of an observer,the superimposing and leading the display image being performed by anoptical system; and

controlling a display size and a display position of the display imageon the display device so that the display image is displayed in an imagesuperimposition region detected from the outside image.

(11) An image display apparatus including:

an optical system that superimposes a display image displayed on adisplay device onto an outside image, and leads the display image to anobserver; and

a display control unit that has a first control mode for performingcontrol so that the display image is displayed in a region on which aline of sight of the observer concentrates in the outside image, and asecond control mode for performing control so that the display image isdisplayed in a region outside the region on which the line of sight ofthe observer concentrates in the outside image.

(12) The image display apparatus of (11), wherein the display controlunit performs control in the first control mode when the observer is notmoving, and performs control in the second control mode when theobserver is moving.

(13) The image display apparatus of (11) or (12), wherein the displaycontrol unit changes a display state of the image in accordance with astate of a region on which the display image is to be superimposed inthe outside image.

(14) The image display apparatus of any of (11) through (13), wherein

the optical system includes a first optical system that superimposes aleft-eye image displayed on a first display device onto an outside imageand leads the left-eye image to the left eye of the observer, and asecond optical system that superimposes a right-eye image displayed on asecond display device onto the outside image and leads the right-eyeimage to the right eye of the observer, and

the display control unit controls disparities of the left-eye image andthe right-eye image so that a depth position of a stereoscopic image tobe perceived by the observer through the left-eye image and theright-eye image is located closer to the front side than a depthposition of a region in which the stereoscopic image is displayed in asuperimposed manner in the outside image.

(15) An image display method including the steps of:

superimposing a display image displayed on a display device onto anoutside image, and leading the display image to an eye of an observer,the superimposing and leading the display image being performed by anoptical system; and

selectively performing control to display the display image in a regionon which a line of sight of the observer concentrates in the outsideimage, and control to display the display image in a region outside theregion on which the line of sight of the observer concentrates in theoutside image.

(16) An image display apparatus including:

an optical system that superimposes a display image displayed on adisplay device onto an outside image, and leads the display image to aneye of an observer; and

a display control unit that changes a display state of the display imagein accordance with a state of a region on which the display image issuperimposed in the outside image.

(17) The image display apparatus of (16), wherein the display controlunit acquires the state of the region in the outside image based oncaptured image data obtained by forming the outside image.

(18) An image display method including the steps of:

superimposing a display image displayed on a display device onto anoutside image, and leading the display image to an eye of an observer,the superimposing and leading the display image being performed by anoptical system; and

changing a display state of the display image in accordance with a stateof a region on which the display image is superimposed in the outsideimage.

(19) An image display apparatus including:

a first optical system that superimposes a left-eye image displayed on afirst display device onto an outside image, and leads the left-eye imageto the left eye of an observer;

a second optical system that superimposes a right-eye image displayed ona second display device onto the outside image, and leads the right-eyeimage to the right eye of the observer; and

a display control unit that controls disparities of the left-eye imageand the right-eye image so that a depth position of a stereoscopic imageto be perceived by the observer through the left-eye image and theright-eye image is located closer to the front side than a depthposition of a region in which the stereoscopic image is displayed in asuperimposed manner in the outside image.

(20) An image display method including the steps of:

superimposing a left-eye image displayed on a first display device ontoan outside image, and leading the left-eye image to the left eye of anobserver, the superimposing and leading the left-eye image beingperformed by a first optical system;

superimposing a right-eye image displayed on a second display deviceonto the outside image, and leading the right-eye image to the right eyeof the observer, the superimposing and leading the right-eye image beingperformed by a second optical system; and controlling disparities of theleft-eye image and the right-eye image so that a depth position of astereoscopic image to be perceived by the observer through the left-eyeimage and the right-eye image is located closer to the front side than adepth position of a region in which the stereoscopic image is displayedin a superimposed manner in the outside image.

REFERENCE SIGNS LIST

-   100, 100A to 100C Head mount display-   101, 101L, 101R Glass lens unit-   102 Connecting member-   103, 103L, 103R Infrared sensor-   104 Gyro sensor-   105, 105L, 104R Camera-   106 Distance measuring sensor-   111, 111L, 111R Display driver-   112, 112L, 112R Display-   121 Eye position estimating unit-   122 Line-of-sight estimating unit-   123 Depth/structure estimating unit-   123C Structure estimating unit-   124 Display control unit-   125 To-be-displayed image generating unit

The invention claimed is:
 1. An image display apparatus, comprising: anoptical system configured to superimpose a display image, displayed on adisplay device, onto a real-world image; and a display control unitconfigured to temporal smooth, by a plurality or coordinate filters, andto control a size and a position of the display image displayed on thedisplay device so that the display image is displayed in an imagesuperimposition region in the real-world image.
 2. The image displayapparatus according to claim 1, wherein the image superimposition regionis detected based on captured image data obtained by the real-worldimage.
 3. The image display apparatus according to claim 1, wherein thedisplay control unit is configured to control the size and the positionof the display image based on image data to display the display image onthe display device based on information about the image superimpositionregion.
 4. The image display apparatus according to claim 1, wherein thedisplay control unit is further configured to change a display state ofthe display image based on a state of the image superimposition regionin the real-world image.
 5. The image display apparatus according toclaim 4, wherein the display control unit is further configured tocorrect image data to display the display image based on the state ofthe image superimposition region so that an element of the real-worldimage is removed from the display image to be observed by an observer.6. The image display apparatus according to claim 1, wherein, based on adetermination that the image superimposition region is undetected fromthe real-world image, the display control unit changes manners ofdisplay of the display image.
 7. The image display apparatus accordingto claim 1, wherein the display control unit is configured to controlthe size and the position based on the temporal smoothing on sizes andpositions on the display device, the sizes and positions are determinedby the image superimposition region that is cyclically detected.
 8. Theimage display apparatus according to claim 1, wherein, in an event achange in the real-world image is detected, the display control unitchanges manners of display of the display image.
 9. The image displayapparatus according to claim 1, wherein the optical system includes afirst optical system configured to superimpose a left-eye imagedisplayed on a first display device onto the real-world image, and asecond optical system configured to superimpose a right-eye imagedisplayed on a second display device onto the real-world image, andwherein the display control unit is further configured to controldisparities of the left-eye image and the right-eye image so that adepth position of a stereoscopic image to be perceived by an observerthrough the left-eye image and the right-eye image is located closer toa front side than a depth position of a region in which the stereoscopicimage is displayed in a superimposed manner in the real world image. 10.The image display apparatus according to claim 1, wherein the pluralityof coordinate filters include average filter, majority filter orinfinite impulse response (IIR) filter.
 11. An image display method,comprising: superimposing a display image displayed on a display device,onto a real-world image; and temporal smoothing, by a plurality ofcoordinate filters, and controlling a size and a position of the displayimage displayed on the display device so that the display image isdisplayed in an image superimposition region in the real-world image.12. An image display apparatus, comprising: an optical system configuredto superimpose a display image, displayed on a display device, onto areal-world image; and a display control unit that includes a firstcontrol mode in which the display image is displayed in a region onwhich a line of sight of an observer concentrates in the real-worldimage, and a second control mode in which the display image is displayedin a region outside the region on which the line of sight of theobserver concentrates in the real-world image, wherein the displaycontrol unit is configured to temporal smooth, by a plurality ofcoordinate filters, and to control a size and a position of the displayimage display on the display device.
 13. The image display apparatusaccording to claim 12, wherein in the first control mode the observer isstable, and in the second control mode the observer is moving.
 14. Theimage display apparatus according to claim 12, wherein the displaycontrol unit is further configured to change a display state of theimage based on a state of a region on which the display image is to besuperimposed in the real-world image.
 15. The image display apparatusaccording to claim 12, wherein the optical system includes a firstoptical system configured to superimpose a left-eye image displayed on afirst display device onto the real-world image, and a second opticalsystem configured to superimpose a right-eye image displayed on a seconddisplay device onto the real-world image, and wherein the displaycontrol unit is further configured to control disparities of theleft-eye image and the right-eye image so that a depth position of astereoscopic image to be perceived by the observer through the left-eyeimage and the right-eye image is located closer to a front side than adepth position of a region in which the stereoscopic image is displayedin a superimposed manner in the real-world image.
 16. An image displaymethod, comprising: superimposing a display image, displayed on adisplay device, onto a real-world image; controlling display the displayimage in a region on which a line of sight of an observer concentratesin the real-world image, and display of the display image in a regionoutside the region on which the line of sight of the observerconcentrates in the real-world image; and temporal smoothing, by aplurality of coordinate filters, and controlling a size and a positionof the display image displayed on the display device.
 17. An imagedisplay apparatus, comprising: an optical system configured tosuperimpose a display image, displayed on a display device onto areal-world image; and a display control unit configured to change adisplay state of the display image based on a state of a region on whichthe display image is superimposed in the real-world image, wherein thedisplay control unit is further configured to temporal smooth, by aplurality of coordinate filters, and to control a size and a position ofthe display image displayed on the display device.
 18. The image displayapparatus according to claim 17, wherein the display control unit isfurther configured to acquire the state of the region in the real-worldimage based on captured image data obtained by the real-world image. 19.An image display method, comprising: superimposing a display image,displayed on a display device, onto a real-world image; changing adisplay state of the display image based on a state of a region on whichthe display image is superimposed in the real-world image; and temporalsmoothing, by a plurality of coordinate filters, and controlling a sizeand a position of the display image displayed on the display device. 20.An image display apparatus, comprising: a first optical systemconfigured to superimpose a left-eye image, displayed on a first displaydevice, onto a real-world image; a second optical system configured tosuperimpose a right-eye image, displayed on a second display device,onto the real-world image; and a display control unit configured tocontrol disparities of the left-eye image and the right-eye image sothat a depth position of a stereoscopic image to be perceived by anobserver through the left-eye image and the right-eye image is locatedcloser to a front side than a depth position of a region in which thestereoscopic image is displayed in a superimposed manner in thereal-world image, wherein the display control unit is further configuredto temporal smooth, by a plurality of coordinate filters, and to controla size and a position of the left-eye image displayed on the firstdisplay device and the right-eye image displayed on the second displaydevice.
 21. An image display method, comprising: superimposing aleft-eye image, displayed on a first display device, onto a real-worldimage; superimposing a right-eye image, displayed on a second displaydevice, onto the real-world image; controlling disparities of theleft-eye image and the right-eye image so that a depth position of astereoscopic image to be perceived by an observer through the left-eyeimage and the right-eye image is located closer to a front side than adepth position of a region in which the stereoscopic image is displayedin a superimposed manner in the real-world image; and temporalsmoothing, by a plurality of ordinate filters, and controlling a sizeand a position of the left-eye image displayed on the first device andthe right-eye image displayed on the second display device.